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Cognitive  science  has  developed  a  number  of  experimental 
paradigms  that  appear  promising  as  tools  for  assessing  expertise. 
However,  they  typically  involve  verbal  report  data  and  much  labor  by 
human  analysts.  This  project  will  automate  some  of  these  paradigms  and 
integrate  them  into  a  prototype  system  for  assessment  of  an  individual’s 
progress  in  achieving  expertise  in  a  particular  task  domain.  We  have 
chosen  college  physics  as  the  task  domain.  Not  only  is  this  task  domain 
rich  in  conceptual  as  well  as  procedural  knowledge,  but  many  expert- 
novice  experiments  used  physics  as  the  task  domain,  so  we  can  build  in  the 
results  of  these  studies,  which  allows  us  to  concentrate  on  automating  their 
measurement  and  analysis. 

From  the  students’  point  of  view,  the  OLAE  (On-line  Assessment  of 
Expertise)  system  is  a  workbench  for  studying  examples  and  solving 
problems  which  occasionally  interrupts  to  ask  for  an  explanation  of  one’s 
action.  OLAE  presents  standard  physics  examples  and  problems,  as  well  as 
qualitative  physics  problems,  problem  classification  tasks,  and  other  tasks. 
The  student  will  attend  lectures  and  read  the  prose  parts  of  the  textbook 
offline,  but  all  the  rest  of  the  student’s  work  will  be  done  on  OLAE.  - 


From  the  assessors’  point  of  view,  OLAE  provides  three  resources  for 
making  decisions.  It  contains  a  huge  store  of  analyses  and  data  on  one  or 
more  students’  performance.  It  provides  facilities  for  further  analyses  of 
these  data.  It  provides  control  over  the  collection  of  further  data. 

OLAE  supports  three  levels  of  analysis:  (1)  The  raw  behavioral  data 
can  be  saved  so  that  it  is  always  available  for  future  analyses.  (2)  A 
computationally  sufficient  representation  of  the  student’s  knowledge,  such 
as  a  rule-based  system,  can  be  inferred.  (3)  Broad  classifications,  such  as 
“understands  Newton’s  second  law’’  or  “learns  rapidly  but  forgets  rapidly,” 
can  be  defined  and  applied.  Most  of  the  classifications  are  based  on  the 
rules,  which  are  in  turn  based  on  the  student’s  performance. 

The  classifications  are  used  both  by  the  assessor  to  make  decisions 
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and  by  this  system  to  control  its  assessment  strategy.  In  general,  the 

system  will  only  collect  and  analyze  data  if  the  resulting  analysis  could 
affect  classifications  that  the  assessor  considers  important.  However,  the 
assessor  can  also  indicate  specific  assessment  policies  for  OLAE  to  follow. 
For  instance,  OLAE  might  always  ask  the  subject  to  estimate  the  difficulty 
of  a  problem  before  solving  it  even  though  the  data  from  that  task 
currently  have  no  bearing  on  classifications  that  the  assessors  has  flagged 
as  important. 

The  measures  that  we  have  selected  for  inclusion  in  OLAE  are  listed 
below. 

1.  OLAE  collects  a  trace  of  the  student’s  goals  and  actions  during 

problem  solving.  These  data  are  used  to  infer  which  rules  the  student  uses 
reliably,  unreliably  or  not  at  all.  This  kind  of  assessment  is  common  in 
intelligent  tutoring  systems. 

2.  As  students  solve  problems,  they  can  refer  to  examples.  OLAE 

records  which  example  the  student  referred  to,  which  lines  were  read  and 

for  how  long.  These  data  are  used  to  infer,  for  instance,  whether  the 

student  is  lost  and  looking  for  a  basic  approach  to  the  problem  or  whether 
she  has  a  basic  approach  and  is  seeking  advice  on  a  specific  detail.  Chi, 
Bassok,  Lewis,  Reimann  and  Glaser  (1989)  have  found  that  different  ways 
of  referring  to  the  examples  correlates  with  how  well  the  students 
understood  them. 

3.  The  students’  speed  and  accuracy  during  problem  solving  will  be 
monitored.  These  form  a  crude  index  of  expertise. 

4.  After  the  student  has  read  a  problem,  OLAE  may  ask  her  to  estimate 
how  long  it  will  take  her  to  solve  it  as  well  as  what  features  of  the  problem 
cause  difficulty.  Chi,  Glaser  and  Rees  (1982)  found  that  expertise  is 
correlated  with  accuracy  in  such  estimates. 

5.  On  some  problems,  the  student  will  be  asked  to  describe  her  basic 
approach  to  solving  the  problem  instead  of  actually  solving  it.  Typed 
descriptions  will  be  analyzed  for  specific  keywords  that  indicate 
knowledge  of  appropriate  problem  solving  schemas.  (The  same  analysis 
will  be  used  for  the  “feature”  given  in  estimating  difficulty.)  Chi,  Glaser 
and  Rees  (1982)  showed  that  ability  to  state  a  problem-specific  basic 
approach  is  correlated  with  expertise. 


6.  The  student  will  be  asked  to  cluster  problems  according  to  their 
similarity,  then  describe  each  cluster  with  a  list  of  short  phrases.  Chi, 
Feltovitch  and  Glaser  (1981)  showed  that  experts  sort  problems  on  the 
basis  of  their  solution  method,  whereas  novices  use  surface  features  to 
assess  similarity. 

7.  As  the  student  studies  an  example  for  the  first  time,  OLAE  will 
record  the  time  spent  examining  each  line  and  how  often  she  looked 
backwards  at  lines  preceding  the  current  line.  From  these  data,  and 
perhaps  some  direct  questions  to  the  student,  OLAE  will  assess  whether 
the  student  is  self-explaining  the  examples.  Chi  et.  al  (1989)  and  others 
have  shown  that  self-explanation  correlates  with  effective  learning. 
VanLehn,  Jones  and  Chi  (in  prep.)  have  shown  that  self-explanation  data 
can  be  interpreted  at  the  rule  level. 

8.  As  the  student  reads  an  example,  OLAE  may  force  the  student  to 
indicate  after  each  line  whether  or  not  she  understood  the  line.  Chi  et  al. 
(1989)  and  others  have  shown  that  accuracy  during  example  reading 
correlates  with  effective  learning. 

9.  The  student  m’y  be  given  qualitative  physics  problems,  such  as 

sketching  the  trajecti...y  of  a  pendulum  bob  after  the  pendulum’s  string  is 
cut,  or  labelling  points  along  the  trajectory  of  a  cannonball  with  the  forces 
and  accelerations  acting  on  it.  Physics  educators  consider  this  an 
important  measure  of  physics  understanding  even  though  it  is  not  reliably 
correlated  with  assessments  of  formal,  quantitative  problem  solving  skill. 
Ploetzner,  VanLehn  and  Chi  (in  prep.)  first  implemented  a  system  to 
determine  a  student’s  qualitative  rules  on  the  basis  of  her  answers  to 
qualitative  questions,  then  showed  that  the  rule  bases  used  to  answer 
qualitative  problems  did  not  overlap  much  with  those  used  for  answering 
formal  problems.  Analyses  in  Chi  and  VanLehn  (1991)  suggest  that 
success  in  formal  quantitative  problems  may  be  attributed  to  the 
acquisition  of  “technical  knowledge,’’  such  as  vector  decomposition, 
whereas  solving  qualitative  problems 

often  involve  “conceptual  knowledge”  and  “knowledge  of  principles.”  Thus, 
qualitative  problems  assess  a  different  skill  than  the  one  taught,  which  is 
nonetheless  important  to  assess. 

As  of  the  end  of  September,  1991,  a  user  interface  for  studying 


examples  and  solving  problems  has  been  implemented.  It  contains  a  “poor 
man’s  eyetracker”  that  allows  OLAE  to  tell  which  lines  of  an  example  the 
student  is  looking  at.  All  the  examples  and  problems  in  the  Chi  et  al. 
(1989)  study  have  been  entered,  and  a  pilot  subject  has  been  run. 
Meanwhile,  a  analysis  scheme  based  on  belief  nets  has  been  sketched.  Its 
computational  feasibility  is  being  explored  by  converting  the  knowledge 
base  of  an  existing  physics  problem  solver.  Cascade,  into  its  terms. 
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